Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Block Verification Feature #414

Merged
merged 19 commits into from
Jan 10, 2025
Merged

feat: Block Verification Feature #414

merged 19 commits into from
Jan 10, 2025

Conversation

AlfredoG87
Copy link
Contributor

@AlfredoG87 AlfredoG87 commented Dec 17, 2024

Description:

New Classes:

  • verification package
    • BlockVerificationStatus (Enum)
    • StreamVerificationHandlerImpl (Implementation)
    • VerificationConfig (record)
    • VerificationInjectionModule (interface)
    • VerificationResult (Record)
  • verification.hasher package
    • CommonUtils (utility class for hashing)
    • Hashes (Record)
    • StreamingTreeHasher (Interface)
    • NaiveStreamingTreeHasher (implementation)
    • ConcurrentStreamingTreeHasher (implementation)
  • verification.service package:
    • BlockVerificationService (Interface)
    • BlockVerificationServiceImpl (implementation)
    • NoOpBlockVerificationService (implementation)
  • verification.session pacakge:
    • BlockVerificationSession (Interface)
    • AbstractBlockVerificationSession
    • BlockVerificationSessionSync
    • BlockVerificationSessionAsync
    • BlockVerificationSessionFactory
    • BlockVerificationSessionType
  • verification.signature package:
    • SignatureVerifier (Interface)
    • SignatureVerifierDummy (Implementation)

Other notable changes:

  • Added StreamVerificationHandlerImpl as a new subscriber to the unverified ring buffer.
  • Changes to the dashboard
  • Changes to the Metrics definitions to add the new metrics
  • Added new Config type to the application
  • Fixed some Unit Tests
  • Added enough coverage for the new classes added.
  • Added Config Documentation to the Docs
  • Added New Dashboard to K8 Helm Chart deployment

Related issue(s):

Fixes #375
Fixes #374
Fixes #376
Fixes #173
Fixes #423
Fixes #424

Notes for reviewer:

For the scenarios below, I use a 501 blocks sample with up-to-date (Dec-18-2024) very high TPS (10K).

The default implementation will be ASYNC that not only is faster, but is also non-blocking

Stats with SYNC implementation:
image

Stats with ASYNC implementation:
image

Checklist

  • Documented (Code comments, README, etc.)
  • Tested (unit, integration, etc.)

@AlfredoG87 AlfredoG87 force-pushed the verification-handler branch 2 times, most recently from 50fbeb9 to fa09df7 Compare December 18, 2024 04:39
@AlfredoG87 AlfredoG87 self-assigned this Dec 18, 2024
@AlfredoG87 AlfredoG87 added Block Node Issues/PR related to the Block Node. New Feature A new feature, service, or documentation. Major changes that are not backwards compatible. labels Dec 18, 2024
@AlfredoG87 AlfredoG87 added this to the 0.3.0 milestone Dec 18, 2024
@AlfredoG87 AlfredoG87 force-pushed the verification-handler branch 2 times, most recently from 932b207 to bd9fb75 Compare December 18, 2024 22:32
@AlfredoG87 AlfredoG87 marked this pull request as ready for review December 18, 2024 23:11
@AlfredoG87 AlfredoG87 requested a review from a team as a code owner December 18, 2024 23:11
Copy link
Contributor

@georgi-l95 georgi-l95 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a fast first review, good progress, left some questions.

Copy link
Contributor

@ata-nas ata-nas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlfredoG87 really good job!

This is my first pass through, I was focused mostly at looking at things on the surface level. Will need additional passes to get into the logic in depth, but the change is very big.

That being said, I comment you some nits/cleanup I found on the first pass.

I have not looked at the metrics and charts json changes as well as tests.

Here general comments:

  • I would not use var ever anywhere, even in test/loops etc. I think it makes things very unreadable and we should not be using that IMO. Not a blocker, but an opinion and a suggestion.

  • I am sure I have missed places, but let's be adding the non-null annotation for the static code analysis.

  • Also I am sure I have missed some places, but lets always have a require non null checks on things that must not be null and we only do assignment (not calling a method on the target object). Lets fail early on these places. (Mostly in constructors or in methods where we need a non null object but we use it after executing a lot of logic before hand)

My focus on the next runs will be to get into and test the logic. Then tests (I presume there will be changes made, so I am hesitant to go to tests yet)

Copy link
Contributor

@ata-nas ata-nas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another look, still no look at tests.

Copy link
Contributor

@ata-nas ata-nas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much improved @AlfredoG87. It seems that some changes that you made (based on previous pr comments) were reverted for whatever reason?! (I believe this was unintentional) You can take a look at previous pr comments from previous passthroughs to see more reverted ones.

@AlfredoG87 AlfredoG87 force-pushed the verification-handler branch from cdd9ba9 to 1c1928b Compare January 6, 2025 18:32
@AlfredoG87
Copy link
Contributor Author

Did a forcce push in order to resolve a merge conflicts.

@jsync-swirlds
Copy link
Member

Did a forcce push in order to resolve a merge conflicts.

With this many commits; perhaps a squash would be in order...

Copy link
Member

@jsync-swirlds jsync-swirlds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only part way through, but wanted to get these comments published.
A lot is suggestions or future considerations, but some are more immediately relevant.

@AlfredoG87 AlfredoG87 force-pushed the verification-handler branch 2 times, most recently from c11b4a6 to 9545e93 Compare January 8, 2025 00:06
Copy link

codecov bot commented Jan 9, 2025

Codecov Report

Attention: Patch coverage is 92.96117% with 29 lines in your changes missing coverage. Please review.

Project coverage is 95.92%. Comparing base (c303fa0) to head (e63fbf1).

Files with missing lines Patch % Lines
...fication/hasher/ConcurrentStreamingTreeHasher.java 89.47% 5 Missing and 5 partials ⚠️
...k/server/verification/hasher/HashingUtilities.java 83.92% 9 Missing ⚠️
.../verification/hasher/NaiveStreamingTreeHasher.java 88.23% 2 Missing and 2 partials ⚠️
...er/verification/StreamVerificationHandlerImpl.java 95.12% 2 Missing ⚠️
...rver/verification/VerificationInjectionModule.java 60.00% 2 Missing ⚠️
...erver/verification/hasher/StreamingTreeHasher.java 60.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main     #414      +/-   ##
============================================
- Coverage     96.78%   95.92%   -0.86%     
- Complexity      409      505      +96     
============================================
  Files            82      100      +18     
  Lines          1460     1867     +407     
  Branches         90      131      +41     
============================================
+ Hits           1413     1791     +378     
- Misses           34       56      +22     
- Partials         13       20       +7     
Files with missing lines Coverage Δ
...a/com/hedera/block/common/utils/MathUtilities.java 100.00% <100.00%> (ø)
...a/com/hedera/block/common/utils/Preconditions.java 100.00% <100.00%> (ø)
.../block/server/config/BlockNodeConfigExtension.java 100.00% <ø> (ø)
...era/block/server/config/ConfigInjectionModule.java 88.88% <100.00%> (+1.38%) ⬆️
...er/config/ServerMappedConfigSourceInitializer.java 100.00% <ø> (ø)
...era/block/server/metrics/BlockNodeMetricTypes.java 100.00% <100.00%> (ø)
...com/hedera/block/server/notifier/NotifierImpl.java 100.00% <ø> (ø)
...a/block/server/pbj/PbjBlockAccessServiceProxy.java 100.00% <ø> (ø)
...a/block/server/pbj/PbjBlockStreamServiceProxy.java 91.22% <100.00%> (+0.15%) ⬆️
.../persistence/storage/PersistenceStorageConfig.java 100.00% <ø> (ø)
... and 18 more

@AlfredoG87 AlfredoG87 force-pushed the verification-handler branch from 3de75a6 to 7e88367 Compare January 9, 2025 22:11
Next steps: Add metrics to measure time to hash.

Signed-off-by: Alfredo Gutierrez <[email protected]>

initial metrics dashboard and observations

Signed-off-by: Alfredo Gutierrez <[email protected]>

some simplifications and moving around code, also more logs for tests added. and temporary stuff.

Signed-off-by: Alfredo Gutierrez <[email protected]>

Some more improvements and moving things to async model. however I think I will create several implementations to the interfaces with some simple and synchronous and another one concurrent and asynchronous.

Signed-off-by: Alfredo Gutierrez <[email protected]>

clean-up halfway.

Signed-off-by: Alfredo Gutierrez <[email protected]>

major refactor, clean-up 80%

Signed-off-by: Alfredo Gutierrez <[email protected]>

added NoOp implementation for the VerificationService

Signed-off-by: Alfredo Gutierrez <[email protected]>

added javadoc to BlockVerificationService interface

Signed-off-by: Alfredo Gutierrez <[email protected]>

some more javadocs

Signed-off-by: Alfredo Gutierrez <[email protected]>

adding some extra exception handling.

Signed-off-by: Alfredo Gutierrez <[email protected]>

cleanup

Signed-off-by: Alfredo Gutierrez <[email protected]>

cleanup

Signed-off-by: Alfredo Gutierrez <[email protected]>

Fixing Existing Unit tests and removing an addition (requires org.checkerframework.checker.qual;) on module-info by accident

Signed-off-by: Alfredo Gutierrez <[email protected]>

Unit test for StreamingTreeHasher and BlockVerificationService and some refactor around BlockVerificationService

Signed-off-by: Alfredo Gutierrez <[email protected]>

VerificationInjectionModuleTest Unit test and spotless improvements

Signed-off-by: Alfredo Gutierrez <[email protected]>

More unit tests

Signed-off-by: Alfredo Gutierrez <[email protected]>

More unit tests, for BlockVerificationSessionSync

Signed-off-by: Alfredo Gutierrez <[email protected]>

More unit tests, for BlockVerificationSessionSync

Signed-off-by: Alfredo Gutierrez <[email protected]>

Adding UT Coverage for BlockVerificationSession related classes.
Small improvements in other tests as well.
Improvements to the flow of finalizeVerification thanks to issues surfaced during unit testing

Signed-off-by: Alfredo Gutierrez <[email protected]>

restoring original simulator properties file

Signed-off-by: Alfredo Gutierrez <[email protected]>

improvements on performance for ASYNC

Signed-off-by: Alfredo Gutierrez <[email protected]>

improvements for the dashboard and the config class so it logs the actual/final configuration values

Signed-off-by: Alfredo Gutierrez <[email protected]>

Added env mappings for new verification config properties

Signed-off-by: Alfredo Gutierrez <[email protected]>

Added some missing javadocs

Signed-off-by: Alfredo Gutierrez <[email protected]>

adding env config properties documentation

Signed-off-by: Alfredo Gutierrez <[email protected]>

Added remaining javadocs across the whole project

Signed-off-by: Alfredo Gutierrez <[email protected]>

style fixes

Signed-off-by: Alfredo Gutierrez <[email protected]>

convert sha384HashTag to final

Signed-off-by: Alfredo Gutierrez <[email protected]>

some extra tests and dashboard improvement, also update the dashboard on the chart deployment

Signed-off-by: Alfredo Gutierrez <[email protected]>

remove extension

Signed-off-by: Alfredo Gutierrez <[email protected]>

style fixes

Signed-off-by: Alfredo Gutierrez <[email protected]>

changing VerificationResult for SIGNATURE_INVALID a more descriptive one. instead of something more ambiguous why not something more explicit.

Signed-off-by: Alfredo Gutierrez <[email protected]>

added missing return statement

Signed-off-by: Alfredo Gutierrez <[email protected]>

adding Objects.requireNonNull checks, not sure if is really needed and not just redundant. in any case, improvement on existing class

Signed-off-by: Alfredo Gutierrez <[email protected]>

adding Objects.requireNonNull checks, not sure if is really needed and not just redundant. in any case, improvement on existing class

Signed-off-by: Alfredo Gutierrez <[email protected]>

removed ExecutorService CommonPool fulfiller for all executor requests on DI, and provided it only for the current verification session factory.

Signed-off-by: Alfredo Gutierrez <[email protected]>

fixing test after

Signed-off-by: Alfredo Gutierrez <[email protected]>

Refactor VerificationConfig to use Preconditions class to handle verifications, and also used MathUtilities to handle the actual verification.

Signed-off-by: Alfredo Gutierrez <[email protected]>

fixing test

Signed-off-by: Alfredo Gutierrez <[email protected]>

using long instead of Long

Signed-off-by: Alfredo Gutierrez <[email protected]>

PR Review changes

Signed-off-by: Alfredo Gutierrez <[email protected]>

PR Review changes

Signed-off-by: Alfredo Gutierrez <[email protected]>

style fix

Signed-off-by: Alfredo Gutierrez <[email protected]>

add Objects.requireNonNull on constructor

Signed-off-by: Alfredo Gutierrez <[email protected]>

Adding missing NonNull

Signed-off-by: Alfredo Gutierrez <[email protected]>

returning to how it was

Signed-off-by: Alfredo Gutierrez <[email protected]>

Simplifying Constructor logic

Signed-off-by: Alfredo Gutierrez <[email protected]>

Feedback and spotless updating the licence

Signed-off-by: Alfredo Gutierrez <[email protected]>

addressing more feedback from PR Review

Signed-off-by: Alfredo Gutierrez <[email protected]>

Improving and defining a log format.

Signed-off-by: Alfredo Gutierrez <[email protected]>

setting the logging.properties as part of the java args so it gets picked up before anything is logged.

Signed-off-by: Alfredo Gutierrez <[email protected]>

mapping the logging.properties file to the docker-compose stack, so is easy to change the properties once the container is created by changing the file on build/docker and restart the container.

Signed-off-by: Alfredo Gutierrez <[email protected]>

removed no longer needed field

Signed-off-by: Alfredo Gutierrez <[email protected]>

changing to the way it can be overriden using env variables

Signed-off-by: Alfredo Gutierrez <[email protected]>

adding non null to dummy class dependencies

Signed-off-by: Alfredo Gutierrez <[email protected]>

spotless apply changes

Signed-off-by: Alfredo Gutierrez <[email protected]>

add a missing @nonnull

Signed-off-by: Alfredo Gutierrez <[email protected]>
… and solving conflicts

Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
…oring an illegal state.

Signed-off-by: Alfredo Gutierrez <[email protected]>
…oring an illegal state.

Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
@AlfredoG87 AlfredoG87 force-pushed the verification-handler branch from aa6c04f to 45d5941 Compare January 10, 2025 16:54
ata-nas
ata-nas previously approved these changes Jan 10, 2025
Copy link
Contributor

@ata-nas ata-nas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Nice work.

Copy link
Contributor

@mattp-swirldslabs mattp-swirldslabs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, I left a few comments

Signed-off-by: Alfredo Gutierrez <[email protected]>
Signed-off-by: Alfredo Gutierrez <[email protected]>
@AlfredoG87 AlfredoG87 dismissed stale reviews from mattp-swirldslabs and ata-nas via 975cbc9 January 10, 2025 18:50
Copy link
Contributor

@mattp-swirldslabs mattp-swirldslabs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@AlfredoG87 AlfredoG87 merged commit 09b617b into main Jan 10, 2025
11 checks passed
@AlfredoG87 AlfredoG87 deleted the verification-handler branch January 10, 2025 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Block Node Issues/PR related to the Block Node. New Feature A new feature, service, or documentation. Major changes that are not backwards compatible.
Projects
None yet
5 participants